[SPARK-59185][SQL] Derive a StartsWith prefix filter from leading-literal LIKE patterns by david-mollitor-db · Pull Request #58484 · apache/spark

david-mollitor-db · 2026-09-02T18:30:14Z

What changes were proposed in this pull request?

LikeSimplification rewrites simple LIKE patterns into cheaper predicates ('A%' ->
StartsWith, '%B' -> EndsWith, 'A%B' -> length guard + StartsWith + EndsWith,
'%B%' -> Contains, exact string -> EqualTo). Multi-wildcard patterns that have a
leading literal but match none of those shapes -- e.g. 'A%B%', 'AB%CD%EF', 'A_B%' --
fall through unchanged and remain a full regex Like, so the data source receives no
predicate and the per-row regex runs on every row.

This PR makes LikeSimplification additionally derive the leading literal A as the
necessary condition StartsWith(col, A), keeping the original LIKE as the exact residual:

col LIKE 'A%B%'  ==>  StartsWith(col, A) && (col LIKE 'A%B%')

StartsWith is placed first so the cheap check short-circuits the regex. The derivation is
restricted to binary-equality collations (StringType.supportsBinaryEquality, i.e.
UTF8_BINARY) and a TreeNodeTag on the residual Like keeps the rule idempotent under the
fixed-point optimizer batch. Only StartsWith is derived from the leading literal; the
LikeAll/LikeAny paths are unchanged.

Why are the changes needed?

Pushdown / pruning. On UTF8_BINARY, StartsWith translates to
sources.StringStartsWith, which prunes Parquet row groups via min/max statistics. Readers
cannot prune on the raw Like, so today a leading-literal multi-wildcard LIKE reads every
row group.
Cheaper per-row evaluation. The StartsWith short-circuits the more expensive regex on
rows that fail the prefix.
Results are unchanged. StartsWith(A) is implied by LIKE 'A%...', and the exact LIKE
is retained as the residual, so the conjunction accepts exactly the same rows. This mirrors
how PostgreSQL, SQL Server and SQLite turn a leading-literal LIKE into a sargable prefix
predicate plus a residual recheck.

The derivation is gated on binary equality for correctness as well as benefit: under a
collation-aware collation (e.g. UTF8_LCASE) the Like regex match (Java regex case flags)
and StartsWith (CollationSupport) can disagree, so StartsWith(A) would not be a sound
necessary condition; and StringStartsWith is pushed down only for UTF8_BINARY (non-binary
is wrapped as CollatedStringStartsWith, which readers ignore). EndsWith/Contains are not
derived from trailing/inner literals because Parquet's StringEndsWith/StringContains have
canDrop = false (no pruning).

Does this PR introduce any user-facing change?

No. Query results are identical; this is a performance improvement (added pushdown and a
short-circuit conjunct on an otherwise unsimplified LIKE).

How was this patch tested?

New unit tests in LikeSimplificationSuite covering: derivation for 'a%b%', a multi-char
prefix with multiple wildcards, '_' patterns, no derivation when there is no leading literal,
no derivation when the pattern contains the escape char, no derivation for a non-binary
(UTF8_LCASE) collation, and idempotency.
A new end-to-end test in ParquetFilterSuite verifying that LIKE 'ab%cd%' / 'ab%cd%ef' /
'a_b%' push a StringStartsWith and prune Parquet row groups.
Updated the Python data source filter-pushdown tests (test_python_datasource and its Connect
parity) and the DataSource.pushFilters docstring to reflect that a leading-literal
multi-wildcard LIKE now pushes a StringStartsWith prefix filter.
build/sbt 'catalyst/testOnly *LikeSimplificationSuite' and
build/sbt 'sql/testOnly *ParquetV1FilterSuite -- -z "leading-literal"' pass; scalastyle clean.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 4.8

david-mollitor-db · 2026-09-08T19:21:01Z

@uros-b You've been so kind with your time. Thanks. Would you mind taking a look at this one too?

uros-b

Thank you @david-mollitor-db for the PR! cc @stevomitric with further review here

uros-b · 2026-09-09T08:58:30Z

-        simplifyLike(input, pattern.toString, escapeChar).getOrElse(l)
+        val patternStr = pattern.toString
+        simplifyLike(input, patternStr, escapeChar)
+          .orElse(derivePrefixStartsWith(input, patternStr, escapeChar, l))


This duplicates input (StartsWith(input, prefix) && Like(input, ...)). LikeAll already refuses that unless CollapseProject.isCheap(child) (SPARK-40228). Please gate this derive path the same way — otherwise rand() LIKE 'a%b%' evaluates two different Rand values, and expensive kids (e.g. sha2) run twice. 'a%' stays single-eval today; this extends duplication to 'a%b%', 'a_b%', etc. A sibling of the existing SPARK-40228 cheap-child test would lock it in.

@uros-b Thanks for the call out! I have also identified another code path that lacked this gate: #58663

@stevomitric Updated. I have also identified another code path that lacked this gate: #58663

stevomitric · 2026-09-09T16:16:18Z

+    if (!binaryCollation || !CollapseProject.isCheap(input) || pattern.contains(escapeChar) ||
+        like.containsTag(LIKE_PREFIX_GUARDED)) {
+      None
+    } else {
+      val prefix = pattern.takeWhile(c => c != '%' && c != '_')
+      if (prefix.isEmpty || prefix.length == pattern.length) {


this bails whenever the escape char appears anywhere in the pattern, but the leading literal can be escape-free while the escape only appears later, e.g. 'ab%c%d%', whose prefix ab is clean.

we could have something like:

if (!binaryCollation || like.containsTag(LIKE_PREFIX_GUARDED)) { None } else { val prefix = pattern.takeWhile(c => c != '%' && c != '_') if (prefix.isEmpty || prefix.length == pattern.length || prefix.contains(escapeChar)) {

…eral LIKE patterns `LikeSimplification` rewrites simple `LIKE` patterns into cheaper predicates (`'A%'` -> `StartsWith`, `'%B'` -> `EndsWith`, `'A%B'` -> length guard + `StartsWith` + `EndsWith`, `'%B%'` -> `Contains`, exact -> `EqualTo`). Multi-wildcard patterns with a leading literal that match none of those shapes -- e.g. `'A%B%'`, `'AB%CD%EF'`, `'A_B%'` -- fall through unchanged and stay a full regex `Like`, so the data source receives no predicate and the per-row regex runs on every row. This derives the leading literal `A` as the necessary condition `StartsWith(col, A)` and keeps the original `LIKE` as the exact residual: col LIKE 'A%B%' ==> StartsWith(col, A) && (col LIKE 'A%B%') `StartsWith` is placed first so the cheap check short-circuits the regex, and it is a predicate the existing pushdown path understands: on UTF8_BINARY it translates to `StringStartsWith`, which prunes Parquet row groups via min/max. Results are unchanged -- `StartsWith(A)` is implied by `LIKE 'A%...'` and the exact `LIKE` is retained as the residual. The derivation is restricted to binary-equality collations (`supportsBinaryEquality`): under a collation-aware collation the `Like` regex match (Java regex case flags) and `StartsWith` (`CollationSupport`) can disagree, so `StartsWith(A)` would not be a sound necessary condition; and `StringStartsWith` only pushes down for UTF8_BINARY. Only `StartsWith` is derived from the leading literal -- Parquet's `StringEndsWith` and `StringContains` do not prune -- and the `LikeAll`/`LikeAny` paths are unchanged. A `TreeNodeTag` on the residual `Like` keeps the rule idempotent under the fixed-point batch. This also updates the Python data source filter-pushdown test and the `DataSource.pushFilters` docstring, which previously documented such patterns as pushing no filters. Generated-by: Claude Opus 4.8

david-mollitor-db force-pushed the like-prefix-startswith-pushdown branch from d399e01 to a09a8a2 Compare September 8, 2026 13:58

uros-b reviewed Sep 9, 2026

View reviewed changes

david-mollitor-db force-pushed the like-prefix-startswith-pushdown branch from a09a8a2 to d8bcfcc Compare September 9, 2026 15:03

stevomitric reviewed Sep 9, 2026

View reviewed changes

david-mollitor-db force-pushed the like-prefix-startswith-pushdown branch from d8bcfcc to ac56b5c Compare September 10, 2026 18:57

Navigation Menu

Uh oh!

[SPARK-59185][SQL] Derive a StartsWith prefix filter from leading-literal LIKE patterns - #58484

[SPARK-59185][SQL] Derive a StartsWith prefix filter from leading-literal LIKE patterns#58484
david-mollitor-db wants to merge 1 commit into
apache:masterfrom
david-mollitor-db:like-prefix-startswith-pushdown

david-mollitor-db commented Sep 2, 2026 •

edited

Loading

Uh oh!

david-mollitor-db commented Sep 8, 2026

Uh oh!

uros-b left a comment

Uh oh!

uros-b Sep 9, 2026

Uh oh!

stevomitric Sep 9, 2026

Uh oh!

david-mollitor-db Sep 9, 2026

Uh oh!

david-mollitor-db Sep 9, 2026

Uh oh!

Uh oh!

Uh oh!

stevomitric Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

david-mollitor-db commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

david-mollitor-db commented Sep 8, 2026

Uh oh!

uros-b left a comment

Choose a reason for hiding this comment

Uh oh!

uros-b Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

stevomitric Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

david-mollitor-db Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

david-mollitor-db Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

stevomitric Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

david-mollitor-db commented Sep 2, 2026 •

edited

Loading